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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment See 37 CFR 1.704(b). 

Status 

1)S Responsive to communication(s) filed on 12 May 2004 . 
2a)[3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-3.5-11.13-19 and 21-24 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) |EI Claim(s) 1-3.5-11.13-19 and 21-24 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
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20 Certified copies of the priority documents have been received in Application No. . 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
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DETAILED ACTION 



Claim Rejections - 35 USC § 103 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

1 . The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 

USPQ 459 (1966), that are applied for establishing a background for determining 

obviousness under 35 U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating 
obviousness or nonobviousness. 



2. Claims 1, 2, 6, 8, 7, 9, 10,14, 15, 17,18 & 22 - 24 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Alshawi (U.S. Patent 581 ,666) in view of 
Throckmorton et al. (US 5818441). 



Regarding claims 1, 9 & 17, Alshawi discloses method/computer readable 
medium having stored instructions with at least one processor for providing real- 
time subtitles [captioning] in an AV signal. The disclosure includes the automatic 
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conversion of an audio [including speech] signal in the AV signal. Alshawi 
describes in Fig 1., a video-based communications device (5,8). The device 
provides segmentation of an AV signal (16) and the further processing of the 
audio [speech] portion of the signal to provide continuous speech-to-subtitles 
[speech-to-text] translation (19,21,22) that has the ability to overlay and display 
text subtitles onto AV signal in real-time [captioning](26). Alshawi does not 
disclose synchronizing the caption data with one or more cues in the AV signal. 
However, Throckmorton et al. teach that a data synchronizing sub-system whose 
function is to synchronize the primary data stream generated by sub-system 10 
with specific associated data. The input to data synchronizing sub-system 20 
is scene information from the primary data stream in the form of timecodes and 
time durations [cues], and data from associated data generator sub-system 16. It 
creates a so called script for the delivery and display of associated data at 
specific points in time. The ability to synchronize the associated data with the 
primary has many benefits including helping the hearing impaired viewers to 
better understand AV content. 

Therefore, it would have been obvious to one of ordinary skill at the time of the 
invention to modify Alshawi with the synchronization of the caption data with one 
or more cues in the AV signal as taught by Throckmorton et al. since it would 
have enhanced the viewing experience of the hearing impaired. 
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Regarding claims 2,10 and 18, Alshawi discloses a method and apparatus that 
captures an AV signal and further provides the audio [speech] portion of the 
signal for conversion to text. Alshawi describes a videophone receiver that has 
an input signal that comprises a camera that represents the visual component of 
the communication and a microphone that represents the audio component of 
the signal that have been encoded. (Col 2, 33 - 40). In addition, Alshawi 
describes an audio/video decoder that accepts an AV input and separates the 
signal into two entities, video signal and audio signal (Col 2, 51 - 55). 

Regarding claims 6, 14 and 22, Alshawi discloses a display that shows at least 
the video and text [caption] data. Alshawi describes simultaneously displaying 
the sending party's video overlaid with real-time subtitles [caption] that translates 
the sender's speech (Col 3, 26 - 29). 

Regarding claim 7 and 23, Alshawi discloses an integrated method and 
apparatus for providing real-time subtitles [captioning] in an AV signal. The 
disclosure includes the automatic conversion of an audio [including speech] 
signal in the AV signal to text [caption] data and associating the audio and text 
[caption] data at a time that corresponds to the video signal wherein the signal 
combination processing system synchronizes the caption data with one or more 
cues in the AV signal. Alshawi describes in Fig 1 ., a video-based 
communications device (5,8). The device provides segmentation of an AV signal 
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(16) and the further processing of the audio [speech] portion of the signal to 
provide continuous speech-to-subtitles [speech-to-text] translation (19,21,22) that 
has the ability to overlay and display text subtitles onto AV signal in real-time 
[captioning](26). 

Regarding claims 8, 15 and 24, Alshawi discloses a method and apparatus for 
translating speech and caption into a second language. Alshawi describes an 
embodiment where the textual signal is translated into a target language that is 
then overlaid onto the video signal as real-time subtitles [caption] (Col 3, 46). 



1 . Claims 5, 13 and 21 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Alshawi (U.S. Patent 581,666) in view of Throckmorton et al. (US 5818441) as 
applied to claims 1, 9 & 17 above, and further in view of Angell et al. (US6513003).. 

Regarding claim 5,13 and 21 , Alshawi does not show the embedding [encoding] 
of the text [caption] data within the AV signal. Instead, a subtitle generator 
(24, Fig. 1 ) is used to overlay text data onto the AV signal. However, Angell et al. 
teach the embedding [encoding] of the text [caption] data within the AV signal 
(Fig 1 (1 08, 1 40); Col 4, Line 55 - Col 5, Line 1 7). Embedding of text signal by 
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synchronizing and encoding the text with the audio video signal allows the 
composite signal to be played on a conventional display device at any location. 



Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of invention to modify the combination of Alshawi and Throckmorton by 
embedding [associating] text data with AV signal data using a database as taught 
by Angell et al. will improve on the real time dissemination of the composite audio 
video signal with closed caption. 



3. Claims 3, 1 1 & 19 are rejected under 35 U.S.C. 103(a) as being are rejected under 

35 U.S.C. 1 03(a) as being unpatentable over Alshawi (U.S. Patent 581 ,666) in view of 

Throckmorton et al. (US 5818441) and further in view of Bozdagi et al. (U.S. Patent 

A 

6647535). 

Regarding claims 3, 11 and 19, the combination of Alshawi and Throckmorton et 
al. discloses an integrated method and apparatus for providing real-time subtitles 
[captioning] in an AV signal. The disclosure includes the automatic conversion of 
an audio [including speech] signal in the AV signal to text [caption] data and 
associating the audio and text [caption] data at a time that corresponds to the 
video signal. Alshawi describes in Fig 1., a video-based communications device 
(5,8). The device provides segmentation of an AV signal (16) and the further 
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processing of the audio [speech] portion of the signal to provide continuous 
speech-to-subtitles [speech-to-text] translation (19,21,22) that has the ability to 
overlay and display text subtitles onto AV signal in real-time [captioning](26). 

Alshawi does not show a method of converting the audio portion of the signal to 
text data that checks whether the amount of caption data is greater than a 
threshold amount or an expiration time before the process of association occurs. 
Bozdagi et al. show a system and method to enable real-time and near real-time 
storyboarding on the world wide web. Bozagi et al. teach the use of processing a 
multimedia document which summarizes the original video by placing 
representative static images and audio converted to text into a web document for 
viewing (Col 2, 5). In addition, the device can control the number of 
representative images transferred to be displayed by the use of a threshold 
(Col5 ( Line45-55). Also, time is used to check the change in intensity between 
representative images (Col 6, 7). This gives the advantage of greater flexibility 
in viewing multimedia and reduces on the overall demand for bandwidth. 

Therefore, it would have been obvious to one of ordinary skill at the time of the 
invention to modify the combination of Alshawi and Throckmorton et al. by the 
use of parameters such as caption amount and time threshold as taught by 
Bozagi et al. that show the benefits of the association of text and images for 
multimedia documents which may include AV signals. 
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4. Claim 16 is rejected under 35 U.S.C. 103(a) as being unpatentable over Alshawi 
(U.S. Patent 5815196) in view of Throckmorton etal. (5818441) and in further view of 
Kirkland et al. (U.S. Patent 5900908). 

Regarding claim 16, the combination of Alshawi and Throckmorton et al. 
discloses an integrated method and apparatus for providing real-time subtitles 
[captioning] in an AV signal. The disclosure includes the automatic conversion of 
an audio [including speech] signal in the AV signal to text [caption] data and 
associating the audio and text [caption] data at a time that corresponds to the 
video signal. The combination of Alshawi and Throckmorton et al. do not show 
portability and the utilization of the device in the classroom. However, Kirkland 
teaches a method of providing encoding caption data into the program signal. 
The apparatus receives a television signal with various description data including 
caption data (Col 9, 15). The device itself is a set-top box that can be co-located 
with a television [portable] and which can be used for live performances, 
classrooms and other types of presentations (Col 3,29 and CoMO, 60). Devices 
with such features help the handicap or hearing impaired by providing portable 
text or audio services. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of invention to further modify the combination of Alshawi and Throckmorton 
et al. by making it portable for use in such venues as a classroom taught by 
Kirkland in order to improve on the capability of the captioning system for use in 
the classroom. 



Response to Arguments 

1. Applicant's arguments with respect to claims 1 -3, 5-11,13- 19 & 21 -24 have 
been considered but are moot in view of the new ground(s) of rejection. 

Conclusion 



2. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
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the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 



Lewis A Michael 
Examiner 
Art Unit 2655 




